Multi-Task and Transfer Learning with Recurrent Neural Networks

Being able to obtain a model of a dynamical system is useful for a variety of purposes, e.g. condition monitoring and model-based control. The dynamics of complex technical systems such as gas or wind turbines can be approximated by data driven models, e.g. recurrent neural networks. Such methods have proven to be powerful alternatives to analytical models which are not always available or may be inaccurate. Optimizing the parameters of data-driven models generally requires large amounts of operational data. However, data is a scarce resource in many applications, hence, data-efficient procedures utilizing all available data are preferred. In this thesis, recurrent neural networks are explored and developed which allow for data-efficient knowledge transfer from one or multiple source tasks to a related target task that lacks sufficient amounts of data. Multiple knowledge transfer scenarios are considered: dual-task learning, multi-task learning, and transfer learning. Dual-task learning is a particular case of multi-task learning in which multiple tasks are learned simultaneously, thus, allowing to share knowledge among the tasks. Transfer learning implies a sequential protocol in which the target task is learned subsequent to learning one or multiple source tasks. Knowledge transfer is explored and applied in the context of fully and partially observable system identification to improve the model of a little observed system by means of auxiliary data from one or multiple similar systems. Fully observable system identification assumes that the observed system state corresponds to the true state while partial observability implies the observation of either a subset of the state variables or proxy variables. The primary contribution of this thesis comprises a novel recurrent neural network architecture for knowledge transfer which uses factored third-order tensors to encode cross-task and task-specific information within composite affine transformations. This model is investigated in a series of experiments on synthetic